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ABSTRACT 

Traditionally, the test score represented by the 
number of items answered correctly was taken as an indicator of the 
examinee's ability level. Researchers still tend to think that the 
number-correct score is a way of ordering individuals with respect to 
the latent trait. The objective of this study is to depict the 
benefits of using ability estimates obtained directly from 
individuals' response patterns instead of their test scores, 
especially when responses are graded po lychot omous ly . The importance 
of substantive model validation is also discussed. Mathematical 
models are presented to show that the use of the test score instead 
of the response pattern itself in ability or attitude estimation 
will, in general, reduce the accuracy of estimation. The loss of 
accuracy can be especially important when items are scored 
polychotomous ly . It is suggested that ability or attitude estimation 
be made from the response pattern using basic functions developed by 
F. Samejima in conjunction with the graded response model. In doing 
so, substantive model validation is essential. (Contains 2 figures, 1 
table, and 13 references.) (SLD) 
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Traditionally, even before mental test theory was proposed, the test score represented 
by the number of items that are answered correctly was taken as an indicator of the 
examinee’s ability level. Today, researchers still tend to think that the number-correct 
test score is a way of ordering individuals v. ith respect to the latent trait, justifying it 
by saying that the test score is a consistent estimator of ability. This idea has gone far 
enough to use monotone likelihood ratio (e.g., Grayson, 1988; Hyunh, 1994) of the test 
score in the latent trait as a way of evaluating mathematical models. 

Samejima (1969, 1972) proposed the general framework of ordered polychotomous 
responses, and Bock (1972) proposed a nominal model for non-ordered polychotomous 
responses which uncovers implicit orders among the nominal responses. After a couple 
of decades, aided by advanced computer technologies, researchers started using mathe- 
matical models that belong to these two big families of models for their research and also 
for more practical purposes of ability and attitude assessments. The idea of monotone 
likelihood ratio has also been extended to evaluate polychotomous response models. 

In reality, there is no such things as an infinitely long test, and our objective should 
be to make the best use of information provided by a finite number of items in the test 
and estimate the individual’s ability level as accurate as possible. Samejima (1969) has 
shown that, in general, the use of the test score, or any aggregate of response patterns, 
will decrease the amount of test information, which results in loss of accuracy in ability 
estimation. 

An advantage of the use of polychotomous responses over the use of dichotomous 
responses lies in increment in the amount of test information. Generally speaking, how- 
ever, loss of test information by using the test score, and hence loss of accuracy in ability 
estimation, is greater when responses arc graded polychotomously. 

The objective of the present study is to depict benefits of using ability estimates 



obtained directly from individuals’ response patterns instead of their test scores, espe- 
cially when responses are graded polychotomously, and also discuss the importance of 
substantive model validation. 



I. Rationale 



Let 6 be the latent variable which assumes any real number, g (= l,2,...,n) 
denote an item, K g be a polychotomous response to item g and and k g denote its 
realization. The operating characteristic, Pk/(d) , of the discrete item response k g is 
defined as 

= prob. [K, = k, | 0] . (1) 

When ordered polychotomous responses are dealt with, K g and k g in (1) are replaced 
by the graded responses X g and x g (= 0, 1,2, ..., m g ) , respectively. 

Samejima (1973) defined the item response information function, A (0) , such that 

4,(0) = -X log P k ,{0) , (2) 

and the item information function lg(0) was defined as its conditional expectation, 
given 6 , of the item response information function 



4(«) = £[4,(0) I «] = E • (3) 

kg 

Equation (3) includes Birnbaum’s item information function for a dichotomous item 
(Birnbaum, 1968) as a special case. 

Let V be a response pattern, or a sequence of item responses, and v denote its 
realization. Assuming local independence (Lord & Novick, 1968), the operating charac- 
teristic, P v (0) , of a specific response pattern v is given by 

Pv(0) = prob.[V = v\0} = n P kg (0) . (4) 

k 9 ev 
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Thus from (2) and (4) the response pattern information function, I v (0 ) , can be written 
-v,as 

w) = ~~ log = z't.m . (5) 

kg£v 

and the test information function, 1(0) , is denned as the conditional expectation of the 
response pattern information function, given 6 , and from (2), (3), (4) and (5) 

1(0) = E(I v (0) I 0} = W) Pv(0) = £ I g (0) . (6) 

V g = 1 

This amount of test information, or its square root, is used as a local measure of accuracy 
in ability or attitude estimation, when the response pattern is used as the basis of 
estimation. 

Let T be any aggregation of response patterns, and t be its realization. The 
operating characteristic, Pt{Q) , of & specific aggregation t is given by 

p,(0) = . (7) 

vet 

When the aggregates t 5 s are disjoint and each response pattern v belongs to one and 
only one t , as is the case with the test score, the test information function, I*(0) , 
based on such an aggregation is defined by 

'•(«) = £[-^lo g P,(0)|«] £[{^logP t («)|0} 2 ] , (8) 

as opposed to the original test information function, which can be written as 

m = ei-l^iogawio] = pu^iogpwion , 

which is obvious from (5) and (6). Using Cauchy-Schwarz’s inequality, Samejima (1969) 
showed that 

no) < no) . (9) 

Inequality (9) implies that, if polychotomous response categories are more finely clas- 
sified, as exemplified by a 7-point sc? versus a 3- point scale, then in general the amount 
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of test information will be increased. More importantly, since the test score is an aggrega- 
tion of response patterns, except for certain special cases, the amount of test information 
will be decreased if the test score is used as the basis of ability or attitude estimation, 
instead of the response pattern itself: the fact that leads to inaccuracy in the latent trait 
estimation. 

Samejima (1969, 1972) proposed a sufficient condition for an item response to pro- 
vide a unique local or terminal maximum likelihood estimate for any response pattern 
consisting of such item responses. Let Ak g (9) be such that 

= A log P„,(D) ■ (10) 

The sufficient condition is that Ak g {9) be strictly decreasing in 9 with non-negative 
and non-positive values for its two asymptotes, respectively. For brevity, this condition 
has often been called the unique maximum condition, ft is noted from (2) and (3) that 
the first part of this condition can be rephrased, that is, the item response information 
function be positive for all 9 except, at most, at an enumerable number of points where 
it is zero. 

ft has been shown (Samejima, 1969, 1972) that both the normal ogive model and the 
logistic model on the dichotomous response level, and also those on the polychotomous 
response level, satisfy this condition, and so does Bock’s nominal response model. Thus 
in these models the likelihood function based on the response pattern, which is the same 
as P v (9) in (4), has a unique local or terminal maximum for every v G V . Note, 
however, that neither the three-parameter logistic model nor the the three- parameter 
normal ogive model satisfies the unique maximum condition (Samejima, 1972, 1973) 
and thus for some response patterns the likelihood functions have multi-modes. Yen, 
Burket fc Sykes (1991) have shown that multi-modality of the likelihood function occurs 
not infrequently for response patterns that usually come across in empirical data if the 



three-parameter logistic model is used. 

If an aggregate T exemplified by the test score is used instead of the response pattern 
V as the basis of ability or attitude estimation, however, unimodality of the likelihood 
function is not assured, even if the model itself satisfies the unique maximum condition. 
This is obvious from the likelihood function Pt(Q) shown in (7). 

From (4) and (10) it is obvious that the likelihood equation is given by 

gt log P.(«) =£*,(«) = 0 . (11) 

kg€v 

Thus if the model satisfies the unique maximum condition the argorithm for obtaining 
the maximum likelihood estimat 6 V will be easy and straightforward, using the ba- 
sic functions. In this way, without usning any aggregates such as the test score, the 
examinee’s ability can be estimated directly from his/her response pattern. 

II. Test Score with Polychotomous Responses 

In Rasch model (Rasch, 1960), equality in (9) holds if the number-right test score 
is used as T , since it is a sufficient statistic. Thus in this model accuracy in ability 
or attitude estimation will not decrease even if it is based on the number-right test 
score. It has been reported by many researchers, however, that in many cases estimated 
operating characteristics of the correct answer, or item characteristic curves, provide 
distinct discrimination powers for separate items in a test. Thus it will be an unjustified 
approach to adopt Rasch model in such cases even though the model porvides the benefit 
of mathematical simplicity. 

When the discrimination parameter is considered as in the normal ogive and logistic 
models, the number-right test score is no longer a sufficient statistic. Thus inequality 
holds in (9) that results in the loss of accuracy in estimating the latent trait when the 
test score is used as the basis of ability or attitude estimation instead of the response 



pattern itself. Decrease in the amount of test information is illustrated in Figure 1 for 
10 hypothetical, dichotomous test items following the logistic model, whose operating 
characteristic, P g (0) , of the correct answer to item g is given by 



P 3 (0) = 



( 12 ) 



1 + exp[Da fl (0 - b g )\ 

where a g and b g are the discrimination and difficulty parameters, respectively, and 
D is a scaling factor approximately equals 1.7 (Birnbaum, 1968). The values of these 
parameters for the 10 items are shown in the same figure. Considering that the number- 
right test score classifies individuals into only (n + 1) categories, whereas the number 
of possible response patterns is as large as 2" , indicating that if n = 10 these numbers 
are 11 versus 1,024 , decrease in the amount of test information is readily acceptable. 



Insert Figure 1 About Here 



When each item is polychotomously scored, the test score is given by J2 Xg ev x g > 
where x g (= 0,1,... ,m s ) is the graded item score. Thus even if the smallest number, 
m g = 2 , applies for each and every item, the number of possible response patterns 
becomes as large as 3" , whereas that of distinct test scores is only 2n + 1 . This 
implies that if n = 10 there are 59,049 possible response patterns in contrast to 
the 21 test score categories, and, for example, the test score 10 includes as many as 
8, 950 different response patterns. To be more specific, consider two response patterns, 
(2, 2, 2,2, 2, 0,0,0, 0, 0) and (0, 0, 0,0, 1 , 1 , 2,2,2, 2) , with the 10 items arranged in 
the ascending order of difficulty. Should these two response patterns be treated as 
equivalent because they provide Hie same test score 10 ? Common sense tells us that 
these two response patterns that belong to the same test score category may represent 



substantially different latent trait levels. Use of the test score results in the loss of all 
specific information contained in each response pattern, however, and these two response 
patterns are treated as if they were equal. 



Insert Figure 2 About Here 



Figure 2 illustrates decrease in the amount of test information caused b}' the use 
of the test score instead of the response pattern itself, for 10 hypothetical, ordered 
polychotomous items with m g - 2 each, with the second difficulty parameter 0.5 higher 
than the first one for each dichotomous item illustrated in Figure 1. As is expected, a 
greater amount of decrement is seen in Figure 2 than in Figure 1. 

A strength of the general ordered polychotomous model is that m g can be different 
for separate items. In such a case, a single test score contains a greater variety of 
response patterns, and a greater loss of test information is expected. Another strength 
of the general ordered polychotomous response model lies in a variety of configurations 
of item response difficulty parameters within each item and across separate items. For 
example, the distance between the difficulty parameters between x g = 1 and x g = 2 
may be substantially larger than the one between x g = 2 and x g = 3 within a single 
item, and this relationship may be reversed across two separate items. The test score 
ignores such distinct configurations of difficulty parameters, and treat them as if they 
were equal, whereas the likelihood equation (11) based on the response pattern reflects 
such differences as they are. 

Scatter diagrams of the test score for polychotomous response items and an estimate 
of the individual’s latent trait based on the response pattern usually provide a wide 
range of dispersion, especially when m g is large for most items. This indicates that 



the test score for polychotomous response items does not represent individual differences 
with respect to the latent trait. 

It is easily seen from (11) that, as long as the model satisfies the unique maximum 
condition, for a test of n graded items, there u-e only 1 m g + n basic functions. 
From this small number of basic functions, using an appropriate, simple algorithm, 
n<?=i(ra fl + 1) likelihood equations can be produced, and the maximum likelihood es- 
timate, 6 V , of the latent trait is obtained as the solution of the likelihood equation 
(11) for a specific response pattern v . For example, when n = 10 and m g = 2 tor 
all items, there are only 30 basic functions in total, and they provide 0 V for each of 
59,049 response patterns. 

III. Substantive Criteria for Model Validation 










Most researchers are concerned with the goodness of fit of the operating characteristics 
that a mathematical model provides to their set of empirical data as the criterion for 
model validation. Samejima (in press) pointed out the danger of this conventional way 
of model validation, illustrating that two mathematical models that are based on totally 
different rationale can produce practically identical sets of operating characteristics, and 
there will not be any way to decide which model is more suitable if curve fitting is 
exclusively used for model validation. In the same token monotone likelihood ratio of 
the test score cannot be used as a criterion for model validation, since the test score only 
asymptotically converges in probability to the latent trait. 

Samejima (in press) proposed more substantive criteria for model validation such as 
the fitness of the rationale or principle behind the model to our data, additivity intrinsic 
in the model, its natural generalizability *0 a continuous model, satisfaction of the unique 
maximum condition, and orderliness of the modal points of the operating characteristics 
that the model provides. Among others, the rationale on which a specified mathematical 
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model is based should be considered intensively. 

One of the issues that needs to be considered is how to accommodate differences in 
difficulty levels of 'terns in ability estimation. On the dichotomous response level, for 
example, should the correct answer to a more difficult item be credited more than that 
to an easier item, or should failure in answering correctly to an easier item be penalized 
more than failure in answering a more difficult item correctly? Samejima (1969) pointed 
out that symmetric models that provide point-symmetric item characteristic curves with 
(ft 5 , 0.5) as the point of symmetry, which are exemplified by the normal ogive and logistic 
models, have a certain logical problem, because they treat correct answers and incorrect 
answers symmetrically, c\ccommodating both of the above philosophies. Taking the 
normal ogive model as an example, whose operating characteristic of the correct answer 
is given by 

P g {9) = exp[ ~ 2 ~\ du (13) 

where a g and b g are the discrimination and difficulty parameters, respectively, this 
problem was presented and discussed in the original literature (Samejima, 1969, pages 
85-86) cis follows: 

“Now let us consider a situation in which examinees are required to solve n di- 
chotomous items, all of whose discriminating powers are the same, but whose difficulty 
levels are different from one another. For simplicity, let us suppose that n = 5 and 
the five items are denoted by 1, 2, 3, 4, and 5, in the order of easiness. If there are 
five examinees who have tried to solve all the five items but succeeded in only one, and 
each item solved is different from each other, to which of the five examinees should be 
assigned the highest valu» of estimate? The answer to this question may largely depend 
upon subjective judgments or preferences. On the normal ogive model, however, the 
answer is definite, as we can easily observe by following the preceding reasoning. We 
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can arrange the five response patterns in the order of high evaluation as the following: 

( 0 , 0 , 0 , 0 , 1 ) ( 0 , 0 , 0 , 1 , 0 ) ( 0 , 0 , 1 , 0 , 0 ) ( 0 , 1 , 0 , 0 , 0 ) ( 1 , 0 , 0 , 0 , 0 ) 

This fact appears to suggest that the philosophy of scoring underlying the normal ogive 
model is such that an examinee is evaluated as low in the ability tested because he can 
only solve an easy item, while an examinee who has solved a difficult item is evaluated 
as high in the ability tested because of the difficulty of the item solved. 

Now let us consider another instance, if there are another five examinees who have 
also tried to solve the same five items and who have succeeded in four of them, but 
failed in only one, and each item failed is different from each other, to which of these 
five examinees should be assigned the highest value of estimate? On the normal ogive 
model, again we can easily see that these five response patterns are arranged in the order 
of the magnitudes of estimates as follows: 

( 1 , 1 , 1 , 1 , 0 ) ( 1 , 1 , 1 , 0 , 1 ) ( 1 , 1 , 0 , 1 , 1 ) ( 1 , 0 , 1 , 1 , 1 ) ( 0 , 1 , 1 , 1 , 1 ) 

In this case the philosophy of scoring seems to be that an examinee is evaluated as low 
in the ability tested because of the fact that he cannot c ^jv'e even an easy item, while 
an examinee who has failed in a difficult item is evaluated as high in the ability tested 
because it is no proof of his inferiority that he has failed in a difficult item. 

The above two philosophies are, in one sense, contradictory with each other, since the 
principle is completely reversed. That is to say, the difficulty of an item in the former 
instance is treated just as the easiness of an item in the latter instance, and vica versa. 
If we apply the principle in the former case to the latter, the order of evaluation of the 
five response patterns should be reversed, since, for instance, an examinee with response 
pattern (0, 1 , 1, 1, 1) has succeeded in the most difficult four items, while an examinee 
with response pattern (1, 1, 1, 1 ,0) has succeeded in the easiest four items.” 
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A solution of this contradiction can be found in the adoption of a certain asymmetric 



model. Samejima (1972) called the family of models represented by 

1 



Pa(0) = 



1 



(14) 



1 + exp [Da g (9 - b g )] 

with £ g (> 0) as the third parameter the positive exponent family of the logistic model. 
It can be shown that, when £ g > 1 , the model consistently follows the philosophy of 
giving more credit to the success in answering a more difficult item correctly, providing 
the reversed order of 9 V ’s for the response patterns that have only one item score 0 
each while the order of the 9 V ’s for the response patterns having only one item score 1 
each is unchanged. This is illustrated in Table 1 with n = 5 , and with the parameter 
values a g = 1 for all items, b g = —3.0, —1.5, 0.0, 1.5, 3.0 , respectively, and = 2 . 



Insert Table 1 About Here 



In this situation, rationale behind the family of models represented by (14) includes 
the slip ratio delined as an monotone decreasing function of 0 . When it is incorporated 
in the logistic model represented by (11), if the slip function is assumed to be 

1 I 

1 + exp[Da s (0 - b g )] 

which is strictly decreasing in 9 , the third parameter £ g in (14) will become 2 ; if it 
is assumed to be 

i 

i 1 

1 + exp[D<z fl (0 - 6 S )] 

which is also strictly decreasing in 0 , £ g will become 1.5 in (14), etc. 

When £ g < 1 , the model represented by (14) follows the principle that failure in 
answering an easy item correctly be penalized more than failure in answering a more dif- 



ficult item correctly. Rationale behind this family of models, including both of the above 
two situations, is discussed in detail in a separate paper (Samejima, in preparation). 

R . Discussion and Conclusions 

It should be kept in mind that the objective of psychological measurement is to 
estimate the latent trait as accurately as possible making the best use of information 
provided by item responses. The use of the test score instead of the response pattern 
itself in ability or attitude estimation will, in general, reduce accuracy of estimation, and 
the loss of accuracy can be enormous especially when items are scored polychotomously. 
It is strongly suggested that ability or attitude estimation be made directly from the 
response pattern using basic functions given by (10), instead of using the test score as 
the estimate of ability or attitude, or estimating the latent trait from the test score. 

In so doing, substantive model validation is essential, for if we use an inappropriate 
model for our data information provided by the model will be useless, or even harmful. 
Usefulness of models that provide asymmetric operating characteristics for dichotomous 
items, or its expansion to ordered polychotomous items such as the acceleration model 
(Samejima, 1995) should seriously be considered. 

References 



O 

ERLC 



[1] Birnbaum, A. (1968). Some latent trait models and their use in inferring an exam- 

inee’s ability, Contributed chapters in Lord, F. M. and Novick, M. R., Statistical 
theories of mental test scores , Chapters 17-20. Reading, MA: Addison Wesley. 

[2] Bock, R. D. (1972). Estimating item parameters and latent ability when responses 

are scored in two or more nominal categories. Psychometrika , 57, 29-51. 

[3] Grayson, D. A. (1988). Two group classification in latent trait theory: Scores with 

1*2 



14 



monotone likelihood ratio. Psychometrika, 53, 283-292. 

[4] Huynh, H. (1994). A new proof for monotone likelihood ratio for the sum of inde- 

pendent Bernoulli random variables. Psychometrika, 59, 77-79. 

[5] Lord, F. M. & Novick, M. R. (1968). Statistical theories of mental test scores. 

Reading, MA: Addison Wesley. 

[6] Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. 

Copenhagen: Nielson k. Lydiche. 

[7] Samejima, F. (1969). Estimation of ability using a response pattern of graded 

scores. Psychometrika Monograph, No. 17. 

[8] Samejima, F. (1972). A general model for free-response data. Psychometrika Mono- 

graph, No. 18. 

[9] Samejima, F. (1973). A comment on Birnbaum’s three- parameter logistic model in 

the latent trait theory, Psychometrika , 38, 221-233. 

[10] Samejima, F. (1995). Acceleration model in the heterogeneous case of the general 

graded response model. Psychometrika , 80, 549-572. 

[11] Samejima, F. (1996). Acceleration model in the heterogeneous case of the general 

graded response model. Behaviormetrika, (in press). 

[12] Samejima, F. (1996). Virtue of asymmetric item charactristic curves, (in prepara- 

tion). 

[13] Yen, W. M., Burket, G. R. and Sykes, R. C. (1991). Nonunique solutions to the 

likelihood equation for the three-parameter logistic model, Psychometrika, 56, 
39-54. 



NCMK962.TEX 



ERIC 



15 



iO o 
oi 03 



I I I I 



CD 10 o 
cr> CZ> 



10 q o o 
c> t-< . — i oi 



II II II II 



Crt 

3 



i-HC*COX , iOOb-OOOf-i 
-O *0 -O *-0 -O -O -O *-C> t-O -o 



H 

00 

05 

v-T 

P 

00 

05 

0 

H 

oo 

O 

00 



05 

O 

oo 

H 

* 

Q 

oo 



to 

S 

in' 

&3 

H 

oo 



ii 

O 



<N 

00 

ro 



II 

O 

< 

Q 

s 

6 

o 

Q 



H 

oo 

H 

c /5 

« 

O 

oo 

d 

t— ( 

D5 

o 



NOHONfld NOIlVIAIUOdNI ±S31 




< 

l - 

LU 

X 

H 



a 

cd 

do 

O 



8 § 
CO O 

8 2 
O ” 

o *-■ 
o < 
n 




'X> 



The original test information function and the one based on the test score of 
hypothetcal test of 10 dichotomous items following the logistic model. 



O lO O IQ o IQ o LO o xo 
(N-Hr^OOO^-i^-i(N(N 



Pi 

V 

to 

*? 

g 



£ 

*f 

cu 

w 

H 

CO 

cf 

N 

U 

C* 

0 

CO to 

#v Ov 

*0 r 5 

<s o 

1 a 

g « 

« co 
AT H 
°o CO 

«x< 

frf t-T 
<s p 

3* 

<1 O 
H 
oo °2 
" O 
n. hj 



0 \ 



It 

O 

< 

K-f 

Q 

s 

d 

d 

H 



H 

co 

H 

CO 

« 

U 

CO 

d 

S 

o 



II II II II II II l! il ii ii 




iooio<z5ioo*nc3io<z> 

CN CN « — i > — i CD CD CD « — i i — I c^} 



II II II II II II !1 II II II 




NOLLONDJ NOUVI/UHOJNI 1S3JL 



< 

F— 

HI 

X 

H 



8 « 

co o 

8 in 



o 3 



CD 



<3 
-»— < 
O 

0 ) 

o 

CJ 

co 



<0 

HD 

O 

s 

o 



to 

5 ) 

o 



CN 



o 



c 


to 


o 


a 


nD 




0 ) 


£ 


$ 




Cv 

J 2 > 


$ 


<D 




CS 

o 


CN 


o 


II 


J 3 


e?j 




C 2 


nD 


fc 


CJ 




Cv 


JD 


- 4-3 


c 

.2 






CO 


u 


C 


c 


C 

o 

- 4-3 


<- 4 — < 




c 


HD 


.2 


O 


V 3 


HD 


<v 


ctf 


Sh 


£ 


to 


ui 




o 


o 




T“< 


.S 


O 






CO 


- 4-3 


V 


CO 


- 4-3 


O 



<3 

d 

;a 

1 c 

o 

o 

J 3 

Eh 



aj 

O 

-jj 

0 ) 

JDS 

- 4-3 

o 

CL 

>> 



CO 




TABLE 1 



The Maximum Likelihood Estimates of 0 Based on 32 Response Patterns of 5 
Dichotomous Items Following the Normal Ogive Model with the Parameters a g = 1.0 
for All Items and b g = -3.0, -1.5, 0.0, 1.5, 3.0 , Arranged in the Ascending Order. 





Response Pattern 


MLE Theta 


1 


00000 


neg. infinity 


2 


10000 


-1.77109 


3 


01000 


-1.40646 


4 


00100 


-0.83936 


5 


00010 


-0.44235 


6 


00001 


-0.34778 


7 


11000 


-0.24612 


8 


10100 


0.10334 


9 


01100 


0.13035 


10 


10010 


0.66449 


11 


01010 


0.67548 


12 


00110 


0.77745 


13 


10001 


1.06032 


14 


01001 


1.06796 


15 


00101 


1.14590 


16 


11100 


1.25580 


17 


00011 


1.47795 


18 


11010 


1.60421 


19 


10110 


1.63116 


20 


OHIO 


1.63323 


21 


11001 


2.16546 


22 


10101 


2.17644 


23 


01101 


2 . 17729 


24 


10011 


2.27846 


25 


01011 


2.27907 


26 


00111 


2.28672 


27 


11110 


2.76207 


28 


11101 


3.11779 


29 


11011 


3.14533 


30 


10111 


3.14744 


31 


01111 


3.14760 


32 


11111 


pos . infinity 
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